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INTRODUCTION 


Traditionally,  a  human  factors  engineer  allocator  functions 
in  the  design  of  a  system  based  on  a  comparison  between  the  oper¬ 
ator's  and  the  machine's  ability  to  do  the  function  using  some 
form  of  the  popular  Fitts'  list  (Fitts,  1951).  The  function  is 
usually  allocated  to  the  one  that  performs  the  task  the  best. 
This  method  has  served  well  for  most  aviation  systems  until  the 
recent  advent  of  the  automation  revolution  in  cockpit  design 
(Uiener  and  Curry,  1980).  The  Fitts'  list  method  of  allocating 
functions  no  longer  works  when  the  computer's  abilities,  at  least 
in  some  aspects,  are  encroaching  on  those  that  have  traditionally 
been  reserved  for  the  human  operator.  We  now  have  computerized 
expert  systems  that  can  perform  traditional  human  activities, 
such  as  heuristic  reasoning,  in  such  diverse  areas  as  medical 
diagnosis,  chemical  spectral  analysis,  and  geographical  oil 
content  surveys  (Cohen  and  Figenbaum,  1982).  The  US  Air  Force 
has  research  underway  to  develop  an  expert  system  for  a  single 
seat  fighter  cockpit  (Pohlmann  and  Payne,  1986;  Shelnutt, 
Stenerson,  Nelson,  and  Marks,  1986).  What  criteria  can  be  used 
to  allocate  functions  when  human  and  computer  abilities  overlap? 

Morris,  Rouse,  and  Frey  (1984)  have  advocated  a  concept  of 
function  allocation  that  is  adaptive.  That  is,  either  the  human 
operator  or  the  computer  accomplishes  the  function  depending  on 
whether  the  operator  or  the  computer  has  been  allocated  the 
responsibility  for  the  function.  Adaptive  allocation  is  executed 
dynamically  in  real  time  by  either  the  operator  or  the  computer, 


depending  on  who  is  in  charge  of  the  allocation  decision.  Thus, 


we  now  have  a  continuum  for  function  allocation;  from  a  fully 
manual  system  (the  operator  does  everything),  to  a  shared  system 
(dynamic  function  allocation),  to  a  fully  automated  system  (the 
computer  does  everything  and  the  operator  monitors)  (Vickens  and 
Kramer,  1985). 

Depending  on  a  system's  design  constraints,  any  point  on 
thi3  continuum  is  feasible.  In  a  single  seat  fighter  cockpit 
environment,  however,  the  shared  function  allocation  scheme 
appears  to  be  the  best  alternative.  At  the  fully  manual  extreme 
the  current  situation  would  still  exist  where  the  pilot  accom¬ 
plishes  most  tasks,  resulting  in  sometimes  unacceptable  levels  of 
workload  (Putterbaugh  and  Varner,  1981).  Consequently,  to  reduce 
pilot  workload  there  is  a  requirement  for  some  form  of  cockpit 
automation  (Air  Force  Studies  Board,  1982).  However,  at  the 
highly  automated  extreme  there  are  also  problems  of  Bystem 
inflexibility  and  pilot  skill  erosion  (Viener,  1985).  These 
problems  are  already  evident  in  commercial  aviation  (Weiner  and 
Curry,  1980).  Thus,  at  least  in  the  fighter  cockpit  environment, 
we  are  forced  to  use  some  function  allocation  strategy  between 
these  two  extremes.  Dynamic  function  allocation  appears  to  be  a 
possible  solution. 

The  purpose  of  this  research  was  to  see  if  dynamic  function 
allocation  is  a  feasible  alternative  for  fighter  cockpit  design. 
This  was  accomplished  by  imitating  such  a  strategy  in  a  simulated 
single  3eat  fighter.  Specifically,  our  hypothesis  was  that  the 
presence  of  a  simulated  expert  system  that  could  be  allocated 
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tasks  would  significantly  iaprovs  Mission  parforaancs.  Thsrs 
wars  thraa  traataant  conditions  usad  to  investigate  thia  hypothe- 
•isi  1)  autoaatic  control  of  tha  dafansiva  countar  aaasuraa 
(DCH)  by  tha  coaputar,  2)  dynaaic  control  of  tha  DCH  by  tha 
coaputar  at  tha  option  of  tha  subject,  and  3)  a  control  condition 
whara  no  coaputar  aiding  was  availabla. 

Anothar  iaportant  varlabla  to  conaidar  in  tha  application  of 
a  dynaaic  function  allocation  schaaa  is  operator  confidence  in 
tha  ability  of  tha  coaputar  to  perfora  its  assigned  tasks.  If 
there  is  little  or  no  trust,  tha  operator  will  bo  forced  to 
Monitor  tha  coaputar'*  parforaancs.  Thia  would  divert  iaportant 
attantional  rasourcas  that  could  ba  used  in  tha  parforaanca  of 
tha  operator's  own  assigned  tasks  (Horrla  at  al. ,  1984). 
Although  operator  trust  in  tha  coaputar  waa  not  Manipulated  in 
this  axparlaont ,  it  was  subjectively  assessed  and  it  was  hoped  a 
high  level  of  confidence  could  ba  achieved. 

METHOD 

Subjects 

Eighteen  aalo  US  Air  Force  Acadeay  cadets  ranging  in  age 
froa  18-23  eerved  as  voluntary  subjects.  Subjects  were  initially 
screened  baaed  on  their  ability  to  play  tha  video  gaaa  siaulation 
usad  in  this  study.  This  was  dons  in  an  atteapt  to  conceptually 
aatch  tha  subject's  skills  to  thoss  of  highly  trained  Air  Force 
pilots. 


i 
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App«r«fcu» 

Siaulat ion .  A  single  scat  fighter  cockpit  environaent  w<« 
siaulated  using  t ha  P-1S  Strika  Eagle  vidao  fast  davolopod  by 
Micro  Prose.  Tha  |«>a  was  playad  on  a  Kaypro  16  coaputer  with  a 
color  aonitor.  P-15  Strika  Bagla  aiaulataa  savan  air-ta-ground 
alsiion  scanarios  with  incraaaing  lavala  of  difficulty.  In 
addition,  thasa  aissiona  can  ba  playad  at  any  ona  of  four  lavala 
of  challenges  arcade,  rookie,  pilot,  and  aca.  Tha  gaaa  alao  haa 
a  varlaty  of  thraata  (i.e.,  anaay  aircraft  and  haat-aaaking  and 
radar-guidad  aisslles)  to  deal  with  while  flying  the  aissions. 

F-15  Strike  Eagle  uaea  four  priaary  viaual  diaplayat  a  Head 
|  Up  Diaplay  (HUD),  tactical  aituation  diaplay,  radar  electronic 

u 

►  warfare  diaplay,  and  a  pictorial  atoraa  diaplay  (Figure  1).  Tha 

offensive  and  defensive  weapons  include  Electronic  Counter 

i  Measures  (ECM),  flares,  duab  boabs,  and  abort  and  aediua  range 

♦ 

l 

alaalles.  For  this  experiaent  tha  ECM  and  flares  ware  the  only 
capabilities  that  could  be  allocated  to  the  siaulated  expert 
systea.  The  gaae  alao  provides  systeas  lnforaation  on  fuel, 
heading,  airspeed,  and  altitude.  Froa  a  huaan  lnforaation 
proceeding  perspective,  this  gaae  realistically  approxiaatas  tha 
fighter  cockpit  anvlronaant  quite  wall. 

i 

i 

To  further  enhance  the  gaae'a  realise,  it  was  played  in  a 
fixed-base  T-38  cockpit  siaulator  containing  three  color  CRTs. 

i 

The  video  gaae  was  presented  on  tha  CRT  located  in  the  HUD 

I 

position.  The  subjects  controlled  the  gaae  through  a  joystick 

I 


and  coaputer  keyboard  that  had  bean  aounted  in  the  cockpit.  In 


Figure  l.  F-15  Strike  Eagle  Cockpit  Layout 
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addition,  tho  subjects  uaad  a  lighted  push  button  switch  mounted 
on  the  forward  position  of  the  left  aide  panel  to  request 


coaputer  aiding. 


An  experiaenter  perforaed  the  functions  of 


the  expert  systea.  A  second  coaputer  keyboard  located  outside 
the  cockpit,  also  connected  to  the  Kaypro  coaputer,  could  be 
used  to  control  the  gaae  In  an  identical  fashion  aa  the  keyboard 
aounted  in  the  cockpit.  The  experiaenter  controlled  which 
keyboard  operated  the  gaae  by  a  two  position  toggle  switch.  To 
further  enhance  the  deception  that  the  subject  was  using  an 
actual  expert  systea,  a  Texas  Instruaents  Business  Pro  coaputer 
equipped  with  a  TI  Speech  board  and  text-to-speech  capability  was 
used  to  advise  the  subject  when  the  siaulated  expert  systea  had 
deployed  a  flare  or  ECM.  Specifically,  the  phrase  used  for  flare 
deployaent  wasi  "Warning!  Heat  seeking  aissile,  flare  deployed"; 
the  phrase  used  for  ECH  ectlvation  was:  "Warning!  Radar  guided 
aissile,  ECH  activated."  The  experiaenter  could  track  the 
aission  on  a  separate  CRT  and  select  the  correct  DCH  when 
appropriate . 

Two  expe r 1 aen t e r s  siaulated  the  expert  systea  and  each 
worked  with  half  of  the  subjects.  The  exper laenters  were  highly 
trained  in  playing  the  gaae  and  consistently  used  an  optimal 
strategy  (based  on  the  program's  user's  manual  and  experience) 
for  deploying  DCH.  Communications  among  the  TI  computer, 
subject,  and  experimenter  occurred  over  an  intercom  system 
through  microphones  and  headsets. 


si 


Experimental  Design 

The  experimental  paradigm  used  for  this  study  was  a  one 
factor,  wlthln-subjects  design.  The  three  levels  of  the  indepen¬ 
dent  variable  were:  1)  automatic  computer  aiding,  2)  dynamic 
computer  aiding,  and  3)  a  control  condition  where  no  computer 
aiding  was  given.  The  treatment  order  was  counterbalanced  using 
a  balanced  latin  square  design.  The  dependent  variable  was  the 
total  score  for  successfully  completed  missions  during  a  single 
experimental  session.  For  a  mission  to  be  considered  success¬ 
ful  ,  the  subject  was  required  to  destroy  the  primary  target  and 
safely  return  to  home  base. 

Procedures 

Pre-test .  Prior  to  participating  in  the  experiment  all 
subjects  were  tested  in  the  simulator  on  their  ability  to  play 
the  video  game.  Each  subject  was  required  to  successfully 
complete  missions  one,  two,  and  three  at  the  rookie  level  of 
difficulty  before  they  were  asked  to  participate.  The  pre-test 
was  administered  approximately  one  month  prior  to  the  experiment. 
The  subjects  were  told  not  to  play  the  video  game  again  prior  to 
their  participation  in  the  data  collection  portion  of  the  study 
and  all  reported  they  had  complied. 

Experimental  sessions.  Subjects  flew  the  simulation  under 
each  condition  during  three  forty-five  minute  sessions.  A  forty- 
five  minute  session  was  chosen  because:  1)  if  a  subject  flew  all 
possible  missions  perfectly  it  would  take  approximately  forty- 


five  minutes  and  2)  usually  a  subject  was  available  for  only 


fifty  minutes.  Each  subject  flew  each  condition  at  approximately 
the  same  hour  of  the  day  on  different  days  with  at  least  one  rest 
day  between  sessions. 

Sub  iect  briefing.  Prior  to  the  session,  the  subjects  were 
told  which  treatment  they  would  see  that  day.  The  expert  system 
was  then  described  in  a  manner  that  lead  the  subjects  to  believe 
they  were  going  to  be  aided  by  an  actual  computer.  To  further 
enhance  the  deception,  the  experimenter  would  then  use  the  TI 
computer's  text-to-speech  synthesis  to  play  the  experimental 
instructions  to  the  subjects  (See  Appendix  A).  The  subjects  were 
then  reminded  of  the  treatment  condition  and  that  their  specific 
objective  was  to  fly  a  successful  mission  as  quickly  as  possible. 
They  were  told  to  only  attack  threats  that  endangered  the  success 
of  the  mission.  For  a  dynamic  allocation  condition,  the  subjects 
were  also  briefed  on  the  operation  of  the  switch  in  the  cockpit 
used  to  request  aiding  from  the  simulated  expert  system. 

Data  collection.  A  subject  started  each  session  flying 
mission  number  one  at  the  rookie  level  of  difficulty.  If  the 
mission  was  flown  successfully,  the  subject  then  flew  mission 
number  two  at  the  pilot  level.  However,  if  the  mission  was  not 
successful  (i.e.,  crashed,  shot  down,  or  failed  to  destroy  the 
primary  target),  the  subject  repeated  the  same  mission  until  it 
was  completed  successfully.  If  the  subject  completed  these  two 
missions  successfully  they  then  flew  mission  four  also  at  the 
pilot  level  (mission  number  three  was  not  used  because  it  lacked 
air-to-air  threats).  If  the  subject  was  again  successful  they 
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would  be  advanced  to  the  aca  level  and  would  again  fly  missions 
nuaber  two  and  four.  However,  if  the  forty-five  minute  time 
period  for  the  session  had  elapeed,  the  aeaalon  was  ended  at  that 
time.  Only  four  subjects  ware  able  to  complete  the  five  possible 
■issions  during  a  session. 

Upon  completion  of  each  successful  mission  the  total  score 
for  that  Mission  was  displayed  to  the  subject  on  the  HUD  and  was 
recorded  by  the  exper inenter  .  Only  total  score  was  recorded 
because  it  was  found  in  previous  research  that  total  score  was 
the  only  variable  sensitive  to  changes  in  perforaanc*  (Arets, 
Guardi  no  ,  McClain,  and  Porterfield,  1986).  Following  each 
session  the  subjects  were  administered  a  questionnaire  using  a 
five  response  Likert  scale.  A  final  questionnaire  was  given 
following  all  three  treatments.  The  subjects  were  not  told  of 
the  simulated  nature  of  the  expert  system  until  all  data  had  been 
collected.  They  were  then  fully  debriefed  as  to  the  necessary 
nature  of  the  deception  involved. 

RESULTS 

The  performance  data  were  analysed  using  the  HANOVA  for 
repeated  measures  procedure  in  the  SPSS/PC+  statistical  package 
for  microcomputers  (Norusis,  1986).  (Even  though  there  wae  only 
one  dependent  variable  measured  in  this  study  the  HANOVA  proced¬ 
ure  had  to  be  used  because  this  is  the  only  procedure  within 
SPSS/PC-t-  that  allows  a  repeated  measures  model.)  The  results  of 
this  analysis  revealed  there  were  no  significant  performance 
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differences  aaoni  the  three  treatment  conditions,  £.(  2 , 32  )■ .  1  3  , 
£.> .  1 .  Tho  ■•ana  and  standard  dsviatlons  ars  shown  in  Tabla  1. 


Tabls  1 

Dsscriptiva  Statistics  for  Allocation  Hsthods 


Condition 

tL 

££ 

Automatic 

9164 

2797 

Dynamic 

9197 

3327 

Control 

8875 

3063 

Ths  subjective  data  obtained  frost  the  Likert  scale  question¬ 
naires  wars  analyzed  using  a  Chi-Square  goodness  of  fit  tost  for 
a  uniform  distribution  ( Sso  Appendix  B).  Some  of  tbs  more 
interesting  findings  ars  as  follows.  First,  11  of  the  18 
subjects  were  confident  in  the  computer's  ability  to  handle  the 
threats,  X2(4  ,  |f»l  7)«1  3 . 9,  p<*.01.  Second,  all  subjects  preferred 
some  form  of  computer  assistance  with  10  of  the  subjects 
selecting  the  dynamic  mode  as  the  best  X^(2,{La,17)a>9.3,  p<  . 01 . 
Third,  11  of  the  subjects  rated  the  computer's  ability  to  handle 
the  threats  as  either  good  or  very  good  x2(4,M»17)«9.8,  p< .  05 . 
Fourth,  14  of  the  subjects  felt  the  computer's  help  actually 
improved  their  performance  on  tho  missions  x2(4,£v17)b25.8, 
p<.001.  Finally,  in  the  dynamic  mode,  11  of  the  subjects  felt 
they  requested  the  computer's  help  either  often  or  very  often 
X* ( 4 , g«l 7 )«1 4 . 6 ,  p<.001.  In  addition,  it  was  found  that  in  the 
dynamic  condition,  subjects  requested  the  help  of  the  computer 
for  an  average  period  of  16  minutes,  or  approximately  one  third 
of  the  time. 


DISCUSSION 


Sine*  the  subjects  in  this  axperiaant  war*  selected  based  on 
thalr  ability  to  play  F-15  Strike  Eagle,  it  is  probably  not 
surprising  to  find  no  significant  perforaance  differences  asong 
the  three  treataent  conditione.  Only  in  situations  where  the 
operator  is  already  overloaded  and  unable  to  perfora  optiaally 
would  a  difference  be  expected.  In  this  study,  the  subjects 
could  already  play  the  video  gaae  quite  wall  without  any  help. 
Therefore,  the  addition  of  the  siaulated  expert  systea  did  not 
help  to  laprove  perforaance.  Haybe  the  Halted  nature  of  the 
coaputer  aiding  in  this  study  did  not  go  far  enough  to  actually 
aid  perforaance.  However,  since  aost  of  the  subjects  did  prefer 
soae  fora  of  assistance  froa  the  expert  systea  it  can  be  suraiaed 
that  although  perforaance  was  not  affected,  the  subjects  at  least 
thought  it  was  helping.  This  is  borne  out  by  the  questionnaire 
results. 

Soae  of  the  subjects  did  coaaent  that  the  coaputer  helped  to 
reduce  their  perceived  workload.  Although  workload  was  not 
specifically  addressed  in  this  research,  it  appears  the  subjects 
were  willing  to  rely  on  the  expert  systea  to  aake  their  task 
easier.  As  one  subject  coaaentedi  "It  is  definitely  reduced 
workload  having  the  coaputer  handle  flares  &  ECH." 

Huaans  are  quite  flexible  and  can  adapt  to  situations  of 
varying  workload  and  still  perfora  adairably  without  any  perfor¬ 
aance  decreaentsi  so  even  if  there  are  no  perforaance  benefits  to 
be  gained  froa  the  addition  of  an  expert  syetea,  operator 


workload  way  bo  raducod.  Although  thia  atudy  provldaa  no 
aapirical  avidanca,  it  hinta  that  by  uainp  an  axpart  ayataa  to 
aid  tha  operator,  attantional  raiourcai  aay  ba  uaad  aora  affi- 
ciantly.  Yutura  raiaarch  abould  concantrtttT  on  objactlva 
workload  diffarancaa  in  addition  to  parforaanoa  diffarancaa. 

Havar thalaaa ,  in  ordar  for  an  oparator  to  roly  on  an  axpart 
ayataa,  thay  auat  ba  abla  to  truat  tha  coaputar  to  do  ita  Job 
wall.  tfhat  ia  conaidarad  pood  parforaanoa  for  ona  paraon  aay 
not  bOv  daaaad  pood  by  anothor.  At  laaat  in  thia  atady,  Boat  of 
tha  aubjacta  fait  tha  aiaulatad  axpart  ayataa  parforaad  favorab¬ 
ly.  But  not  all  aubjacta  fait  thia  way.  Ivan  thouph  tha  axpart 
ayataa  parforaad  quita  conaiatantly  and  waa  aiaulatad  uainp  only 
two  axpar iaantara  who  wara  hiphly  trainad  axparta  in  playinp  tha 
paaa,  aoaa  of  tha  aubjacta  atill  did  not  lika  tha  way  tha 
"coaputar"  handlad  tha  DCM.  Thia  waa  probably  bacauaa  tha 
aiaulatad  axpart  ayataa  (i.a.,  tha  axpar iaantar )  waa  daployinp 
DCH  uainp  a  diffarant  atratapy  than  tha  aubjact  would  hava 
(Horria  at  al.,  1984).  That  ia,  tha  "coaputor'a  aodal"  of  how 
tha  DCH  ahould  ba  handlad  waa  diffarant  froa  that  of  tha  aub- 
Jact'a.  Thraa  of  tha  aubjacta  coaaantad  that  tha  coaputar  did 
not  raact  quickly  anouph,  avan  though  tha  axpar iaantara  wara 
uainp  an  optiaal  atratapy.  Ona  aubjact  who  ratad  tha  coaputar'a 
aid  aa  not  halpful  at  all  atatad  "Tha  coaputar  waa  way  bahind  on 
flaroa  and  ECH".  But  thia  aaaa  aubjact  atill  aaid  that  "it  [tha 
coaputar]  waa  pood  to  uaa  on  tha  tarpat  approach"  [a  high 
workload  aituation].  Anothar  aubjact  alao  atatad  tha  coaputar 
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could  have  conitrvtd  flares  better  (there  wee  a  Halted  number) 
and  another  subject  had  commented  that  the  computer  waited  too 
long  to  deploy  DCH.  But  still  only  one  subject  rated  the 
computer's  performance  as  poor.  It  would  be  interesting  to  see 
how  manipulation  of  factors  such  as  these  (i.e.,  the  computer's 
decision  making  model)  effect  a  subject's  perceived  performance 
of  the  computer. 

Another  point  that  probably  needs  to  be  made  is  that  at  the 
expert  levels  of  F-15  Strike  Eagle,  it  is  very  difficult  to 
complete  a  mission  successfully,  even  for  a  highly  trained 
player.  Only  three  subjects  rated  their  overall  performance  as 
very  good,  with  11  subjects  rating  themselves  as  good  or  average. 
Therefore,  a  majority  of  the  subjects  perceived  neither  theirs 
nor  the  computer's  performance  as  perfect  at  these  expert  levels. 
Norris  et  al.  (1984)  hypothesized  that  when  an  operator's 
perceived  performance  goes  down,  they  are  more  likely  to  rely  on 
a  computer  aid.  This  may  have  occurred  in  this  experiment.  Even 
though  some  of  the  subjects  did  not  feel  the  computer  performed 
well,  they  still  relied  on  it  during  high  workload  situations, 
the  point  at  which  their  perceived  performance  probably  went 
down . 

CONCLUSIONS 

These  results  support  the  idea  that  in  eystems  where  highly 
trained  operators  are  used  to  perform  several  concurrent  complex 
tasks,  the  eddition  of  an  expert  system  to  provide  some  form  of 
computer  aiding  may  not  Increase  overall  system  performance; 


I 


A 
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unlaaa  tha  taak  la  vary  difficult  for  an  oparator  to  parfora  wall 
in  tha  firat  placa.  Thia  outcoaa  aay  ba  tha  conaaquanca  of  tha 
liaitad  natura  of  coaputar  aiding  uaad  in  thia  atudyi  atill, 
aubjacta  prafarrad  aoaa  fora  of  coaputar  halp.  Tha  aain  banafit 
to  ba  gainad  froa  an  axpart  ayataa  appaara  to  ba  tha  raductlon  of 
oparator  workload.  Nonatholan ,  to  gain  any  advantage  froa 
coaputar  aiding,  thia  atudy  point#  to  an  laportant  variabla  that 
naada  to  ba  conaidarad  —  oparator  truat  in  tha  coaputar. 
Snaci f ically ,  how  tha  oparator'a  own  intarnal  parforaanca  aodal 
aatchaa  that  of  tha  coaputar'a. 
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APPENDIX  A 

Instructions  to  Subjects  for  the  Request  Condition 


& 


Hello.  I  an  your  on  board  intelligent  computer.  Hy  purpose 
is  to  help  you  fly  your  mission  successfully.  During  your 
missions.  I  can  automatically  activate  your  threat  management 
capabilities  when  they  are  required  in  your  request  my  help.  To 
request  my  help  you  must  press  the  "C"  switch  on  the  left  front 
panel  of  the  cockpit.  Our  attack  orders  ere  to  fly  directly  to 
the  target,  attack  the  target,  and  return  directly  to  hone  base. 
Only  attack  the  threats  that  endanger  the  success  of  the  mission. 
If  you  happen  to  get  low  on  fuel,  you  may  have  to  return  to  base 
to  rsfuel  before  completing  your  mission.  If  you  have  any 
questions  ask  the  experimenter  at  this  time. 


18 


APPENDIX  B 

Questionnaire  Results 


FINAL  QUESTIONNAIRE 


SUBJECT: 


Circle  the  appropriate  response: 


1.  Overall,  did  you  find  the  keyboard  very  difficult  to  operate? 

0  16  8  1 
Very  Difficult  Difficult  OK  Easy  Very  Easy 

*P  <-05 

2  .  When  the  computer  was  handling  the  threats,  were  you 
confident  in  its  ability  to  do  a  good  job? 

2  9  4  2 

Definitely  Yes  So-So  No 

*P<-05 

3.  Which  mode  of  computer  assistance  did 
computer  aided  threat  management? 


None  Automatic  Requested 

p  <.01 

4.  Taking  all  three  missions  that  you  flew  together,  how  would 
you  rate  Just  YOUR  performance  at  the  video  game? 


0 

0 

6 

8 

3 

Very  Poor 

Poor 

Average 

Good 

Very  Good 

*  ✓ 

P  <-oi 

5 .  Taking 

the  two  mi 

ssion8  together 

where  the 

computer  handled 

the  threats, 

how  would 

you  rate  Just 

the  COMPUTER 

* S  performance? 

0 

1 

5 

7 

4 

Very  Poor 

Poor 

Average 

Good 

Very  Good 

*P  <-05 


6.  Are  there  any  other  general  comments  you  would  like  to  make: 


Not  at  All 


you  prefer  for  the 


» 


DYNAMIC  CONDITION  QUESTIONNAIRE 


SUBJECT: _ 

Circle  the  appropriate  reipome  below: 


1.  How  would  you  assess  lust  YOUR 

missions? 

performance  on  today's 

0 

3 

5 

6 

2 

Very  Poor 

Poor 

Average 

Good 

Very  Good 

2.  Did  you  find  the  keyboard  very  difficult  to  operate? 

0 

1 

5 

8 

2 

Very  Difficult 

Difficult 

OK 

E  a  8  y 

Very  Easy 

*p<-oi 

3.  How  often 

would  you  say  you  requested 

the  computer's  help? 

2 

9 

3 

2 

0 

Very  Often 

Often 

Some 

Not  Often 

Not  at  All 

*P  <s-05 

4.  When  you 

helpful  ? 

did  request 

the  computer ' 

s  help,  did  you  find  it 

5 

7 

3 

0 

1 

Very  Helpful 
*p  <^.05 

Helpf  ul 

Somewhat  Not 

Helpful 

Helpful 

Not  Helpful 
At  all 

5.  Do  you  think  the  computer  improved 

missions? 

your  performance  on  the 

5 

6 

A  1 

0 

Definitely 

Yea 

So-So  No 

Not  at  All 

, 

P  <  • 1 

6.  Here  you 

threats? 

confident  in 

the  computer  ' 

e  ability 

to  handle  the 

2 

6 

5  2 

1 

Definitely 

Yes 

So-So  No 

Not  at  All 

7  . 


Are  there  any  other  comments  you  would  like  to  make: 
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CONTROL  CONDITION  QUESTIONNAIRE 


SUBJECT: 


Circle  the  appropriate  response  below: 

1.  How  would  you  assess  your  performance  on  today's  missions? 


A 

Poor 


5 

Good 


Very  Poor  Poor  Average  Good  Very  Good 

2.  Did  you  find  the  keyboard  very  difficult  to  operate? 


0  1 
Very  Difficult  Difficult 


Somewhat 


8 

Easy 


Very  Easy 


P  <-05 

3.  Are  there  any  other  comments  you  would  like  to  make: 


4 


Myc 


AUTOMATIC  CONDITION  QUESTIONNAIRE 


SUBJECT : 


Circle  the  appropriate  response  below: 


1.  How  would 

you 

assess  iust  YOUR  pert 

ormance  on  today 

missions? 

0 

1 

8 

7 

0 

Very  Poor 

Poor  Average 

Good 

Very  Good 

%  <  .001 

2.  Did  you  find 

t  he 

keyboard  very  difficult 

to  operate? 

0 

2  4 

6 

4 

Very  Difficult 

Di  f  f 

icult  Somewhat 

Easy 

Very  Easy 

3.  Do  you  think 

the  missions? 

the 

computer's  help 

improved 

your  performance 

3 

11 

0 

2 

0 

Definitely 

Yes 

.mewhat 

No 

Not  at  All 

*p  <.001 

4.  Did  you  find  the  computer's  voice  in  telling  you  what  it  was 
doing  helpful? 

4  7  3  2  0 

Very  Helpful  Helpful  So-So  Not  Helpful  Not  Helpful 

*  y  ,  At  All 

p  V 1 

5.  Were  you  confident  in  the  computer's  ability  to  handle  the 
threats? 

0  9  4  3  0 

Definitely  Yes  So-So  No  Not  at  All 

*  s 

P  <-05 

6.  Are  there  any  other  comments  you  would  like  to  make: 


